{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Metadata Extraction with Fenic\n",
    "\n",
    "This notebook demonstrates how to extract structured metadata from unstructured document text using fenic's semantic operations. We'll explore two different approaches for metadata extraction:\n",
    "\n",
    "1. **ExtractSchema** - Fenic's native schema definition approach\n",
    "2. **Pydantic Models** - Using familiar Python class syntax\n",
    "\n",
    "Both methods leverage large language models to intelligently parse and extract structured information from diverse document types including research papers, product announcements, meeting notes, news articles, and technical documentation.\n",
    "\n",
    "## What You'll Learn\n",
    "\n",
    "- Setting up fenic sessions with semantic capabilities\n",
    "- Creating DataFrames from document data\n",
    "- Extracting structured metadata using AI-powered operations\n",
    "- Comparing different extraction approaches\n",
    "- Understanding the trade-offs between schema types\n",
    "\n",
    "Let's dive in!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and Configuration\n",
    "\n",
    "First, we need to import the necessary libraries and configure our fenic session. We'll set up:\n",
    "\n",
    "- **Type hints** from `typing` for better code documentation\n",
    "- **Pydantic** for our second extraction approach \n",
    "- **Fenic** as our main DataFrame library\n",
    "\n",
    "For the session configuration, we're setting up semantic capabilities using OpenAI's GPT-4o-mini model with specific rate limits:\n",
    "- **RPM (Requests Per Minute)**: 500 requests\n",
    "- **TPM (Tokens Per Minute)**: 200,000 tokens\n",
    "\n",
    "This configuration ensures we can efficiently process our document extraction tasks while staying within API limits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Literal\n",
    "from pydantic import BaseModel, Field\n",
    "import fenic as fc\n",
    "\n",
    "# Configure session with semantic capabilities\n",
    "config = fc.SessionConfig(\n",
    "        app_name=\"document_extraction\",\n",
    "        semantic=fc.SemanticConfig(\n",
    "            language_models={\n",
    "                \"mini\": fc.OpenAIModelConfig(\n",
    "                    model_name=\"gpt-4o-mini\",\n",
    "                    rpm=500,\n",
    "                    tpm=200_000,\n",
    "                )\n",
    "            }\n",
    "        ),\n",
    "    )\n",
    "\n",
    "# Create session\n",
    "session = fc.Session.get_or_create(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Document Data\n",
    "\n",
    "Now let's create our test dataset. We've carefully selected 5 diverse document types to showcase the versatility of metadata extraction:\n",
    "\n",
    "1. **Research Paper** (`doc_001`) - Academic study on neural networks and climate prediction\n",
    "2. **Product Announcement** (`doc_002`) - CloudSync Pro file synchronization software launch\n",
    "3. **Meeting Notes** (`doc_003`) - Engineering team standup with decisions and action items\n",
    "4. **News Article** (`doc_004`) - Breaking news about a data breach incident\n",
    "5. **Technical Documentation** (`doc_005`) - API reference for an authentication service\n",
    "\n",
    "Each document contains different types of metadata (titles, dates, keywords, etc.) that we'll extract automatically. After creating the DataFrame, we'll inspect the basic properties including document IDs and text lengths to understand our data better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents_data = [\n",
    "        {\n",
    "            \"id\": \"doc_001\",\n",
    "            \"text\": \"Neural Networks for Climate Prediction: A Comprehensive Study. Published March 15, 2024. This research presents a novel deep learning approach for predicting climate patterns using multi-layered neural networks. Our methodology combines satellite imagery data with ground-based sensor readings to achieve 94% accuracy in temperature forecasting. The study was conducted over 18 months across 12 research stations. Keywords: machine learning, climate modeling, neural networks, environmental science.\"\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"doc_002\",\n",
    "            \"text\": \"Introducing CloudSync Pro - Next-Generation File Synchronization. Release Date: January 8, 2024. CloudSync Pro revolutionizes how teams collaborate with real-time file synchronization across unlimited devices. Features include end-to-end encryption, automatic conflict resolution, and integration with over 50 productivity tools. Pricing starts at $12/month per user with enterprise discounts available. Contact our sales team for a personalized demo.\"\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"doc_003\",\n",
    "            \"text\": \"Weekly Engineering Standup - December 4, 2023. Attendees: Sarah Chen (Lead), Marcus Rodriguez (Backend), Lisa Park (Frontend), James Wilson (DevOps). Key decisions: Migration to Kubernetes approved for Q1 2024, new CI/CD pipeline reduces deployment time by 60%, API rate limiting implementation scheduled for next sprint. Action items: Sarah to finalize container specifications, Marcus to document database migration plan.\"\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"doc_004\",\n",
    "            \"text\": \"Breaking: Major Data Breach Affects 2.3 Million Users. December 12, 2023 - TechCorp announced today that unauthorized access to customer databases occurred between November 28-30, 2023. Compromised data includes email addresses, encrypted passwords, and partial payment information. The company has implemented additional security measures and is offering free credit monitoring to affected users. Stock prices dropped 8% in after-hours trading.\"\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"doc_005\",\n",
    "            \"text\": \"API Reference: Authentication Service v2.1. Last updated: February 20, 2024. The Authentication Service provides secure user login and session management for distributed applications. Supports OAuth 2.0, SAML, and multi-factor authentication. Rate limits: 1000 requests per hour for standard accounts, 10000 for premium. Available endpoints include /auth/login, /auth/refresh, /auth/logout. Response format: JSON with standardized error codes.\"\n",
    "        }\n",
    "    ]\n",
    "\n",
    "# Create DataFrame\n",
    "docs_df = session.create_dataframe(documents_data)\n",
    "\n",
    "docs_df.select(\"id\", fc.text.length(\"text\").alias(\"text_length\")).show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Method 1: ExtractSchema Approach\n",
    "\n",
    "Our first approach uses **fenic's native ExtractSchema system**. This method provides several advantages:\n",
    "\n",
    "✅ **Complex Data Types**: Supports lists, nested structures, and rich type definitions  \n",
    "✅ **Type Safety**: Built-in type checking and validation  \n",
    "✅ **Native Integration**: Seamlessly works with fenic's DataFrame operations  \n",
    "\n",
    "### Schema Definition\n",
    "\n",
    "We'll define a schema with 5 key metadata fields:\n",
    "- **title**: The main subject or title of the document\n",
    "- **document_type**: Category classification (research paper, news, etc.)\n",
    "- **date**: Any temporal information mentioned\n",
    "- **keywords**: List of important terms and topics *(note: this will be a proper list)*\n",
    "- **summary**: Brief one-sentence overview\n",
    "\n",
    "### Extraction Process\n",
    "\n",
    "The extraction happens in three steps:\n",
    "1. **Apply semantic extraction** using `fc.semantic.extract()` with our schema\n",
    "2. **Flatten the results** by accessing nested metadata fields  \n",
    "3. **Display results** to see the structured output\n",
    "\n",
    "Notice how the `keywords` field will return an actual Python list - this is a key advantage of ExtractSchema!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define schema for document metadata extraction\n",
    "doc_metadata_schema = fc.ExtractSchema([\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"title\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"The main title or subject of the document\"\n",
    "    ),\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"document_type\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"Type of document (e.g., research paper, product announcement, meeting notes, news article, technical documentation)\"\n",
    "    ),\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"date\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"Any date mentioned in the document (publication date, meeting date, etc.)\"\n",
    "    ),\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"keywords\",\n",
    "        data_type=fc.ExtractSchemaList(element_type=fc.StringType),\n",
    "        description=\"List of key topics, technologies, or important terms mentioned in the document\"\n",
    "    ),\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"summary\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"Brief one-sentence summary of the document's main purpose or content\"\n",
    "    )\n",
    "])\n",
    "\n",
    "# Apply extraction using ExtractSchema\n",
    "extracted_df = docs_df.select(\n",
    "    \"id\",\n",
    "    fc.semantic.extract(\"text\", doc_metadata_schema).alias(\"metadata\")\n",
    ")\n",
    "\n",
    "# Flatten the extracted metadata into separate columns\n",
    "extract_schema_results = extracted_df.select(\n",
    "    \"id\",\n",
    "    extracted_df.metadata.title.alias(\"title\"),\n",
    "    extracted_df.metadata.document_type.alias(\"document_type\"),\n",
    "    extracted_df.metadata.date.alias(\"date\"),\n",
    "    extracted_df.metadata.keywords.alias(\"keywords\"),\n",
    "    extracted_df.metadata.summary.alias(\"summary\")\n",
    ")\n",
    "\n",
    "extract_schema_results.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Method 2: Pydantic Model Approach\n",
    "\n",
    "Now let's explore the **Pydantic model approach**, which offers a different set of advantages:\n",
    "\n",
    "✅ **Familiar Syntax**: Uses standard Python class definitions that most developers know  \n",
    "✅ **Rich Validation**: Leverages Pydantic's powerful validation system  \n",
    "✅ **Literal Types**: Can constrain values to specific options (great for categories)  \n",
    "⚠️ **Simple Types Only**: Limited to basic types (str, int, float, bool, Literal)  \n",
    "\n",
    "### Key Differences from ExtractSchema\n",
    "\n",
    "The most important limitation to understand: **complex types like lists must be represented as strings**. This means:\n",
    "- ✅ ExtractSchema: `keywords: List[str]` → `[\"AI\", \"machine learning\", \"climate\"]`\n",
    "- ⚠️ Pydantic: `keywords: str` → `\"AI, machine learning, climate\"`\n",
    "\n",
    "### Model Definition\n",
    "\n",
    "Our Pydantic model includes:\n",
    "- **Literal constraints** for `document_type` to ensure valid categories\n",
    "- **Field descriptions** for better LLM understanding\n",
    "- **String representation** for the keywords list (comma-separated)\n",
    "\n",
    "### Why Use Pydantic?\n",
    "\n",
    "Despite the limitations, Pydantic models are excellent when you need:\n",
    "- **Strict validation** with constrained choices\n",
    "- **Familiar Python patterns** for your team\n",
    "- **Simple data structures** without nested complexity\n",
    "\n",
    "Let's see how it performs on the same documents!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define Pydantic model for document metadata\n",
    "# Note: Pydantic models for extraction support simple data types (str, int, float, bool, Literal)\n",
    "# Complex types like lists must be represented as strings (e.g., comma-separated values)\n",
    "class DocumentMetadata(BaseModel):\n",
    "    \"\"\"Pydantic model for document metadata extraction.\"\"\"\n",
    "    title: str = Field(..., description=\"The main title or subject of the document\")\n",
    "    document_type: Literal[\"research paper\", \"product announcement\", \"meeting notes\", \"news article\", \"technical documentation\", \"other\"] = Field(..., description=\"Type of document\")\n",
    "    date: str = Field(..., description=\"Any date mentioned in the document (publication date, meeting date, etc.)\")\n",
    "    keywords: str = Field(..., description=\"Comma-separated list of key topics, technologies, or important terms mentioned in the document\")\n",
    "    summary: str = Field(..., description=\"Brief one-sentence summary of the document's main purpose or content\")\n",
    "\n",
    "# Apply extraction using Pydantic model\n",
    "pydantic_extracted_df = docs_df.select(\n",
    "    \"id\",\n",
    "    fc.semantic.extract(\"text\", DocumentMetadata).alias(\"metadata\")\n",
    ")\n",
    "\n",
    "# Flatten the extracted metadata into separate columns\n",
    "pydantic_results = pydantic_extracted_df.select(\n",
    "    \"id\",\n",
    "    pydantic_extracted_df.metadata.title.alias(\"title\"),\n",
    "    pydantic_extracted_df.metadata.document_type.alias(\"document_type\"),\n",
    "    pydantic_extracted_df.metadata.date.alias(\"date\"),\n",
    "    pydantic_extracted_df.metadata.keywords.alias(\"keywords\"),\n",
    "    pydantic_extracted_df.metadata.summary.alias(\"summary\")\n",
    ")\n",
    "\n",
    "pydantic_results.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup and Conclusion\n",
    "\n",
    "Finally, we properly close our fenic session to free up resources.\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "After running both approaches, you should notice several important differences:\n",
    "\n",
    "🎯 **When to use ExtractSchema:**\n",
    "- Need complex data structures (lists, nested objects)\n",
    "- Want type-safe operations with native list handling\n",
    "- Prefer fenic's native schema system\n",
    "- Working with hierarchical or deeply structured data\n",
    "\n",
    "🐍 **When to use Pydantic Models:**\n",
    "- Need strict validation with constrained choices (Literal types)\n",
    "- Team is familiar with Pydantic patterns\n",
    "- Working with simple, flat data structures\n",
    "- Want familiar Python class syntax\n",
    "\n",
    "### The Keywords Difference\n",
    "\n",
    "Pay special attention to how each method handles the `keywords` field:\n",
    "- **ExtractSchema**: Returns `[\"keyword1\", \"keyword2\", \"keyword3\"]` (actual list)\n",
    "- **Pydantic**: Returns `\"keyword1, keyword2, keyword3\"` (comma-separated string)\n",
    "\n",
    "This fundamental difference affects downstream processing and is crucial for choosing the right approach for your use case.\n",
    "\n",
    "Both methods successfully extract structured metadata from unstructured text, but they excel in different scenarios. Choose based on your data complexity needs and team preferences!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up\n",
    "session.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
